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ABSTRACT 



As part of its contract to develop a framework for 
continuous school improvement in its four-state region (Kentucky, Tennessee, 
Virginia, and West Virginia) , Appalachia Educational Laboratory staff 
designed the Quest project. Based upon principles of inquiry, collaboration, 
and action research. Quest supports and investigates ongoing school 
improvement efforts through rallies, summer symposia, a Scholars program, 
visits to participating schools, communication via listserv and mailings, and 
the creation of a Quest network of schools. This report describes and 
assesses the first summer symposium, held in Gatliiiburg, Tennessee, on August 
10-11, 1998. Participants included 73 students, teachers, parents, staff, and 
administrators from 11 schools, 4 school districts, 2 universities, and the 
Kentucky State Department of Education. The symposium was more like 
conventional professional development activities than other Quest events, 
offering sessions on senior projects, creating and using rubrics to assess 
student work, project-based learning, student writing portfolios. 
Microsociety, and a process called Interview Design. Evaluation data were 
generated by evaluator participant observation, unstructured interviews, 
final evaluation feedback, and the interview design process. Most 
participants thought the sessions were informative and relevant. The 
knowledge and preparedness of presenters received the highest ratings. 
Recommendations included more clearly addressing the needs of elementary 
educators, more actively engaging attendees, and offering evidence of the 
effectiveness of assessment strategies described in sessions. Appendices 
present session evaluation forms, final evaluation forms, session scores, and 
the evaluation standards checklist. Contains 16 references. (TD) 
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AEL's mission is to link the knowledge from research with the wisdom from practice to improve 
teaching and learning. AEL serves as the Regional Educational Laboratory for Kentucky, Tennessee, 
Virginia, and West Virginia. For these same four states, it operates both a Regional Technology in 
Education Consortium and the Eisenhower Regional Consortium for Mathematics and Science 
Education. In addition, it serves as the Region IV Comprehensive Technical Assistance Center and 
operates the ERIC Clearinghouse on Rural Education and Small Schools. 

Information about AEL projects, programs, and services is available by writing or calling AEL. 




Appalachia Educational Laboratory 
Post Office Box 1348 
Charleston, West Virginia 25325-1348 
304/347-0400 
800/624-9120 
304/347-0487 (Fax) 
aelinfo@ael.org 
http://www.ael.org 



This publication is based on work sponsored wholly or in part by the Office of Educational Research 
and Improvement, U. S. Department of Education, under contract number RJ96006001. Its contents do 
not necessarily reflect the views of OERI, the Department, or any other agency of the U. S. Government. 

AEL is an Equal Opportunity/Affirmative Action Employer. 
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EXECUTIVE SUMMARY 



As part of its contract to develop a framework for continuous school improvement in its four- 
state region, Appalachia Educational Laboratory (AEL) staff designed the Quest project. Based 
upon principles of inquiry, collaboration, and action research. Quest proposes to support and 
investigate ongoing school improvement efforts through bi-annual conferences (which staff renamed 
rallies), summer symposia, a Scholars program, visits to participating schools, communication via 
listserv and mailings, and the creation of a Quest network of schools. This evaluation report 
describes and assesses the first Quest summer symposium, held in Gatlinburg, Tennessee, on August 
10 and 11, 1998. 

Including nine AEL staff — three of whom presented sessions, 73 people attended the 
symposium. Six Quest schools sent at least one member; one sent a team of nine, while another sent 
a team of 1 1 . Five schools not affiliated with the Quest network also sent attendees, sending from 
one to six participants. School teams included parents, students, teachers, principals, assistant 
principals, and aides. Three districts sent from one to three central office staff, while a fourth district 
sent a principal from each of its schools in collaboration with another AEL project. The Kentucky 
State Department of Education sent two participants. Four participants from universities attended 
as well. 

The symposium was evaluated in terms of the degree to which participants found the sessions 
relevant, engaging, and worthwhile; the presenters, knowledgeable and prepared; and the information 
on assessment, thought-provoking. 

Analysis of the data suggested that most participants thought the sessions were informative 
and relevant. The knowledge and preparedness of presenters received the highest mean ratings 
across all sessions. The degree to which presenters actively engaged participants received the lowest 
mean rating, although even this score was quite high. Participant observation, unstructured 
interviews, final evaluation feedback, and data generated by the Interview Design process all 
suggested that participants found the symposium valuable. 

Based upon these data, the evaluator concluded that the symposium had been informative and 
useful to participants. Recommendations included designing sessions that more clearly addressed 
the needs of elementary level educators, ensuring that sessions provided activities that actively 
engaged attendees, and offering evidence of the effectiveness of assessment strategies described in 
symposium sessions. 
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INTRODUCTION 



In the summer of 1996, Quest staff at the Appalachia Educational Laboratory (AEL) began 
working with teams from school communities in three West Virginia county school districts to 
invigorate efforts for continuous school improvement, using a variety of techniques for gathering 
input from all those with a stake in their local schools (Howley-Rowe, 1 998f). This first “learning 
community,” known as Leadership to Unify School Improvement Efforts (LUSIE), was comprised 
of school teams including students, teachers, administrators, parents, and community members. 
Ultimately, this group wrote individual school visions and improvement plans, and coauthored with 
AEL Creating Energy for School Improvement (1997), a supplemental guide for those poised to 
write their own state-mandated school improvement plans. 

Quest staff also planned to create learning communities devoted to exploring continuous 
school improvement across the four-state AEL region. Hence, staff scheduled a pilot Inquiry Into 
Improvement conference in April 1997 for selected regional high schools. Schools were selected 
in several ways. Some schools were recommended for the Quest experience by central office staff 
or school administrators. Other schools were asked to join Quest because they had participated in 
previous AEL programs. Still other schools were invited because staff believed they were primed 
for the kind of collaborative inquiries into school improvement Quest was designed to provide. 

On October 5-7, 1997, in Roanoke, Virginia, another conference was held for designated high 
schools in AEL’s region, this time with an explicit emphasis on forming and nurturing a network 
of high schools (Howley-Rowe, 1998c). The high school network met again in Pipestem State Park 
nearBluefield, West Virginia, on February 8-10, 1998 (Howley-Rowe, 1998d). The first conference 
for elementary schools was held on November 2-4, 1997, in Nashville, Tennessee, intending to 
nurture a network for elementary schools (Howley-Rowe, 1998a). A second elementary “rally,” as 
staff renamed the conferences, was convened on February 22-24, 1998, in Lexington, Kentucky 
(Howley-Rowe, 1998b). 

Quest staff also convened project events during the summer of 1998. On July 16-18, the 
Quest Scholars, a select group of 1 1 network members, met at the AEL offices in Charleston, West 
Virginia, to collaborate with staff in the planning, evaluation, and research components of the project 
(Howley-Rowe, 1998e). And a summer symposium, entitled “Focus on Student Work,” was held 
on August 10-11 in Gatlinburg, Tennessee, for both network members and others interested in 
student assessment and continuous school improvement. This report summarizes evaluation findings 
from this symposium. 

Seventy-three people attended the symposium, including nine presenters (three of whom 
were also Quest network members), four Quest staff, three AEL staff from projects other than Quest, 
and the AEL monitor from the Office of Educational Research and Improvement. Counting the three 
presenters from a Quest school, a total of 59 people not otherwise employed by or monitoring AEL 
were in attendance. Symposium attendees drew from elementary, middle, high, and vocational 
schools. Six Quest schools sent at least one member; one sent a team of nine, while another sent a 
team of 1 1. Five schools not affiliated with the Quest network also sent attendees, sending from one 
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to six participants. School teams included parents, students, teachers, principals, assistant principals, 
and aides. Three districts sent from one to three central office staff, while a fourth district sent a 
principal from each of its schools in collaboration with another AEL project. The Kentucky State 
Department of Education sent two participants. Four participants from two universities attended as 
well. 



The purpose of this evaluation report is to assess participant reaction to the symposium, 
particularly in terms of whether attendees found the various sessions useful, relevant, and interesting. 
This report also will examine participant evaluation of the symposium as a whole. The primary 
audience for this report is Quest staff. It is intended to provide information to staff as they make 
decisions about future project events and the development of the network. In addition, this report 
will be a part of an ongoing series of reports about Quest events (Howley-Rowe, 1 998a-1998f). This 
series will document the evolution of the Quest network and the process whereby staff strive to 
enable continuous school improvement. Consequently, this report also may prove useful to others 
interested in building networks or promoting school improvement over time. 
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METHODOLOGY 

I 

A The methods used for this evaluation component of the Quest project were qualitative and 

I quantitative. In terms of qualitative data collection, the evaluator engaged in participant observation 

of symposium activities (Becker & Geer, 1957; Emerson, 1983; Glazer, 1972; Miles & Huberman, 
H 1 994; Patton, 1 980), a method highly suited “for studying processes, relationships among people and 

M events, the organization of people and events, continuities over time, and patterns” (Jorgensen, 1989, 

p. 12). Furthermore, consistent with the Quest paradigm, participant observation involves “a 
A flexible, open-ended, opportunistic process and logic of inquiry through which what is studied 

M constantly is subject to redefinition based on field experience and observation” (Jorgensen, 1989, 

p. 23). This method “is a commitment to adopt the perspective of those studied by sharing in their 
I ... experiences” (Denzin, 1989, p. 1 56), thereby enabling researchers to evaluate how an event or 

• process appears and feels to participants. By “exploit[ing] the capacity that any social actor 

^ possesses for learning new cultures, and the objectivity to which this process gives rise,” participant 

'f observation further produces data that is both rich and valid (Hammersley & Atkinson, 1 983, p. 8). 

^ During the symposium, the evaluator played a role more akin to what Denzin typifies as 

M “observer as participant,” rather than as a complete participant observer (1989). That is, the 

evaluator’s contact with rally attendees was not as a participant in the activities in which they were 
A engaged, but instead as a roaming onlooker and occasional conversationalist. The evaluator sat in 

I on symposium sessions, watched the larger group convene at the beginning and end of the 

symposium, and took advantage of serendipitous occasions to chat. 

* Because three sessions were scheduled concurrently during each two and a half hour block, 
the evaluator chose to attend only one session during each scheduled block during the first day of 

M the symposium. On the second day, the evaluator pursued this strategy again, with one exception. 

P During a break, the evaluator was invited to watch the second half of a session presented by a Quest 

network school, and so did not return for the end of the session she had originally attended. The 
P strategy of attending only one session per scheduled block meant that limited portions of the 

^ symposium were subject to participant observation. However, it did offer the evaluator the 

^ opportunity to observe sessions in their entirety. In all, the evaluator observed five sessions, three 

H firom beginning to end. 

J| In order to corroborate the theses generated by participant observation, the evaluator also 

I. collected and analyzed quantitative and qualitative data. Using several data sources in order to 

^ corroborate theses is what Brewer and Hunter ( 1 989) call “multimethod research” or “triangulation.” 

H This approach posits that the strengths of each method will compensate for the weaknesses in others, 

P ultimately providing a more complete account of that being studied. 

A Hence, in addition to participant observation, two evaluation forms were used to collect 

further information. The first form was distributed at the end of each o f the six sessions offered (see 
Appendix A). Each session was offered twice over the course of the symposium; however. 
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evaluation forms did not solicit information about which of the two offerings was being evaluated 
by respondents. The evaluator was less concerned with assessment of each offering individually 
than with an overall assessment of the quality and usefulness of the material presented in both 
offerings of the six sessions. A second form was distributed at the end of the symposium, and was 
designed to solicit participant reaction to the entire symposium (see Appendix B). 

Unstructured interviews also were conducted during the course of the symposium. As 
opportunities arose for relatively private conversation, participants were asked to discuss their 
assessments of the symposium, generally, and of the sessions they had attended, specifically. 
Interview responses were later categorized and analyzed by theme. 

Analyses of participant observation field notes, interview data, and evaluation forms were 
made by question, and sometimes by theme. 
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SYMPOSIUM EVENTS 



Participants in the Quest summer symposium filled a conference room at 8:30 a.m. on August 
10, 1998, in Gatlinburg, Tennessee. At registration, they had received a large notebook binder with 
articles and information relating to each session offered, as well as some information about the Quest 
project itself Too, they received a schedule of events, which indicated an introductory session fix>m 
8:30 a.m. to 9:30 a.m. At 8:36, then, the symposium officially began as a Quest staff member 
welcomed the attendees and briefly introduced the project to those unfamiliar with it. She described 
the purposes of the introductory session: “to get to know each other,’ “to take a look at what s 
coming [at the symposium],” and “[This] soimds silly, but to think about school improvement. Not 
just cleanup, new grading— but by looking at student work, at better ways to look at it. I hope you 
will all be thinking about [this].” 

Then the Quest facilitator shared a story from a school improvement network newsletter 
about a group of heart surgeons whose patients’ survival rates had increased substantially after the 
group agreed to observe each other’s work over the course of several months. It s risky to share 
what we do in isolation — ^but we hope to support [each other],” said the facilitator, elaborating on 
the Quest goal to sustain continuous school improvement efforts. 

Next the facilitator requested attendees take part in an activity called Just Like Me. In this 
activity, the session presenters stood facing the audience, each one in turn naming a personal or 
professional experience or attribute they had. Then the audience members who shared such an 
experience or attribute were to stand and say, “Just like me!” Most participants appeared to find this 
activity amusing; many laughed, sometimes leaping to their feet to join in proclaiming the phrase. 

Following this, the facilitator briefly explained the regional educational laboratory program, 
then asked that participants “resist the temptation to shop during our time.” She added that “at 3:58 
tomorrow” (the symposium was scheduled to end at 4:00 p.m. the following day) there would be 
door prizes for those who attended the final whole group session. After reviewing the agenda, the 
facilitator noted that time would be allotted during this introductory period for school teams to 
discuss what sessions to attend. 

Another activity was introduced called Quest Bingo. Attendees were given a bingo card; in 
each space was written a trait or experience (e.g., “Has used project-based learning” or “Is a 
grandparent”). Participants were to find another participant who possessed one of the traits or had 
one of the experiences listed to sign the attendant space. They were to continue this activity until 
they had signatures in all the spaces; attendees who finished first were awarded prizes (e.g., gift 
certificates for educational products, canvas bags with the Quest logo). Most participants seemed 
to enjoy playing the game. They talked loudly and milled about, filling the conference room with 
noise and movement. Attendees made reciprocal exchanges of signatures — “I’ll sign yours if you 
sign mine.” After approximately 15 minutes, the facilitator announced “We have the top six 
winners!” Those still gathering signatures groaned in apparent disappointment. 
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Following the distribution of prizes, the facilitator discussed the Quest project. She noted 
that school improvement is a process, one that is “inside-out and upside-down.” She described the 
three major Quest goals to connect, create, and commit, and the Quest habits of storytelling, 
reflection, and inquiry. In addition, she used the metaphorical depiction of Quest as a tree to help 
illustrate the project, adding that the summer symposia were more like conventional professional 
development events than were the other Quest offerings. 

The facilitator then requested that participants read a statement of the Quest vision for 
schools, which had been included in their notebooks. Attendees were to discuss in small groups — at 
their tables — ^“what idea do you resonate most with [in the vision]?” and “what idea would be most 
challenging for schools?” Following about five minutes of discussion, the facilitator asked if 
participants were willing to share their answers. Several people responded. 

As it was nearing 9:30, the facilitator suggested that attendees spend a few minutes writing 
and reflecting on their experiences and thoughts thus far. Afterwards, school teams were provided 
a few minutes to discuss what sessions to attend. A break was scheduled from 9:30 to 9:45, when 
the first session began. 

Six sessions were offered each day of the s)miposium, three each during two separate two- 
and-a-half hour blocks of time. In this way, all six sessions were offered to participants twice. The 
evaluator was able to observe two sessions on August 10. The first concerned creating and using 
rabrics to assess student work. Twenty-three participants attended this session. After engaging the 
attendees in some discussion about the relationship between four components of quality learning 
(curriculum, instruction, assessment, classroom culture), the presenter initiated the central activity. 
This involved sorting ten student papers into three categories of proficiency. After sorting the 
papers, the presenter asked participants to tell her which papers they had sorted into which 
categories. This questioning revealed that attendees had categorized the papers according to quite 
different criteria. The discussion then turned to building mbrics, using participants’ criteria for 
assessing one paper that had been most often placed in the “excellent” category. An attendee asked 
about the structure of the writing assignment itself, indicating that it was unclear and of ambiguous 
value. The presenter used this question as a means to discuss how student work could be used 
productively for assessment, but only if assignments were clear and focused. 

The next task the presenter asked participants to engage in required the entire group to agree 
on criteria by which to judge the student papers. She requested that attendees begin with the middle 
category of proficiency, asking them to agree on “the essentials” papers must have. The group was 
able to constract criteria for this category, and so then moved on to discussing criteria for assessing 
papers as excellent. Last, the group chose criteria by which to recognize a paper that was not 
proficient. While most of the participants were attentive during this activity, a few were less 
engaged, whispering to each other and ignoring the presenter. The presentation ended with a brief 
comparison of holistic and analytical trait scoring and of generalized and task-specific rubrics. The 
presenter wrapped up her session at 12:20 p.m. 
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After lunch, the evaluator attended a session about senior projects, which began at 1:30. 
Although several more participants arrived during the presentation, only five symposium attendees 
were seated at the beginning of the session. The presenter announced that she was “an evangelist” 
concerning senior projects because she had “seen it work.” Much of her presentation, then, consisted 
of vivid stories testif^ng to various student experiences with senior projects. For instance, she told 
the story of Juana, a working-class Mexican American high school senior raising a small son, whose 
project concerned keeping teenage mothers in school. She told of a Cambodian senior who refused 
to speak, yet wrote a novel about his family’s execution and his own escape for his senior project. 
Her stories seemed to suggest that these projects rendered the final year of public schooling more 
relevant to the students. The presenter also described several ways of conducting the senior project 
process, briefly covering issues such as what department might be responsible for the process and 
the various roles required to implement it. She spoke of senior projects as a “rite of passage” for 
seniors, and as a process of “renewal and validation” for staff. In terms of planning to implement 
senior projects, the presenter suggested “designing down” or “beginmng at the end,” asking what 
a high school diploma ought to mean. She suggested a diploma might indicate that its possessor had 
certain attitudes, skills, and knowledge. These, she said, were acquired through attending classes, 
meeting course requirements, maintaining an adequate GPA, and completing final exams. In turn, 
such activities are supported by “curriculum, curriculum, curriculum.” Finally, curriculum is 
undergirded by a school vision. 

On the second day of the symposium, the evaluator observed three sessions: one in its 
entirety and half each of two others. The first session concerned project-based learning, and was 
scheduled to begin at 8:30 a.m. Participants straggled in, so the presenter waited until 8:43 to 
commence. Twenty-three people ultimately participated in the session. Having presented a session 
the previous day, she announced that she would offer “a quick version of yesterday’s lecture and 
then [have] lots of hands-on” activities. Then the presenter described her experiences with project- 
based learning, offering detailed illustrations of the mles and procedures she relied upon to facilitate 
successful projects. These guidelines included administering instruments to students to assess their 
thinking and conflict resolution styles, and their “intelligences.” Students were then to use such 
information to constmct compatible teams for completing projects. Next, the presenter discussed 
the daily routine she had implemented, with each team doing particular tasks. As she described her 
rules and routines, she also discussed several of the projects her students had completed, fi’om 
creating cars out of mousetraps to designing boxes for soda cans. 

After a break, the presenter talked about contracting with students for grades and constructing 
rubrics. Additionally, she discussed communication with parents about project-based learning and 
the essentials for implementing the process successfully (including lesson plans, timelines, materials, 
and integrated teacher teams). She ended her session with several more examples of projects, 
sharing publications that described them and a Web page her students had created to showcase their 
work. 

The evaluator observed the first half of a session on student writing portfolios following 
the lunch break. Twelve people attended this session. The presenters began with a description of 
the Kentucky Education Reform Act that had, among other things, mandated the assessment of 
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student writing portfolios. Following this, the presenters had participants at each table discuss one 
of three questions: What kind of school achievement test did you take when you were in school? 
How was writing assessed on that test? Can you remember how the results were reported to you and 
your parents? None of the attendees had taken achievement tests that assessed writing. 

Next, the presenters asked participants to read two examples of student writing and jot down 
what they thought made each adequate or not. After some discussion with participaiits about their 
assessments of the two writing samples, the presenters then described the types of wnting required 
in each portfolio. At approximately 1 :30 p.m., the presenters excused attendees for a short break. 

During the break, the evaluator was invited to attend the remaining half of the session on 
Microsociety, a program in which students create and maintain a miniature society, complete with 
banks, businesses, a justice system, and television station. In this session, the presenters both 
discussed Microsociety and participated in the California Protocol process. This process enables 
presenters to receive feedback about their program in a non-threatening manner. When the evaluator 
joined the session, participants were discussing their critiques of Microsociety in small groups. 
Many of the comments were warm, or supportive. Several participants offered their concerns that 
Microsociety was not adequately integrated with academics and that too much time was spent 
conducting the program. One concern a presenter had was that the school was not “capitalizing on 
the hooks” created by Microsociety. 

Following discussion with the entire group about participant reaction to Microsociety, a 
Quest staff member asked the group to discuss their assessment of the California Protocol process 
itself. Presenters and participants alike reported that they found the process useful, noting that it was 
informal; nonthreatening; felt open and honest; allowed the presenters, in addition to the audience, 
to leam; and provided a way to obtain suggestions for improvement. The presenters said that they 
planned to use the process at their school to discuss their work. The session ended at 2:26. 

The closing session was scheduled from 3:00-4:00 p.m., during which attendees participated 
in an activity called Interview Design. Participants were queried about their evaluations of the 
symposium. Two rows of chairs were set up facing one another, one pair of chairs per question. As 
the process began, participants faced each other and took turns asking the question found on the seats 
of their chairs, and recording each other’s answer. Next, the facilitator asked that each participant 
in only one of the rows move to the seat to his or her right, while the participant on the end of the 
row moved to the seat at the beginning of the row. Then the process continued with participants 
asking the new people with whom they found themselves faced the question they had found on their 
original seats. This seat-switching continued until all the questions were asked of all the 
participants. The second phase of this activity involved analyzing the data generated from the 
interviewing process. All of the participants asking a particular question assembled as a group to 
compile their data and look for trends in the answers they received. The final phase of Interview 
Design involved summarizing the data. One representative from each group gave a brief 
presentation of the trends their group had found in the answers they analyzed. 
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As promised at the beginning of the symposium, Quest staff awarded prizes to those who 
remained until the end. Participants wrote their names on slips of paper, which were collected and 
placed in a box. The evaluator drew names from the box, and a staff member presented each winner 
with her or his prize. These included a literacy training videotape and book, and gift certificates to 
a school products catalog. An attendee who had made obvious his submission of several slips of 
paper won the first prize presented, much to the amusement and playful indignation of many. The 
symposium ended amidst clapping and joking. 
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FINDINGS 



This section summarizes findings concerning the symposium from participant observation, 
unstructured interviews, and evaluation forms. 

The session evaluation forms posed eight closed-response statements to participants, using 
a Likert-type scale ranging from “1” (indicating that the participant “highly disagree[d]” with the 
statement) to “5” (indicating that the participant “highly agree[d]” with the statement). Participants 
were also offered a “don’t know” response option (subsequently treated as missing data in the 
analysis). Respondents also were asked two open-ended questions. One asked participants to 
describe the most valuable things they had learned from the session, and the other requested 
suggestions for improving the session. 

Overall, participants reported that the presenters had been knowledgeable and well-prepared 
and that sessions had been worthwhile, relevant, had helped participants think about assessment of 
student work, and had stimulated them to want to learn more about the topics (see Table 1). Across 
all the sessions, the mean score for participant assessment of presenter knowledge was 4.82 (SD .46), 
and the mean score for presenter preparation was 4.75 (SD .46). The statement regarding session 
activities that solicited active engagement from participants received the lowest mean rating across 
all sessions, 4.49 (SD .76). However, this score nonetheless indicates that participants tended to 
agree that session activities engaged them actively. 



Table 1 

Combined Session Evaluations 



Condensed Evaluative Statements 


N Mean 


SD 


1. Relevant and useful information 


185 


4.60 


.62 


2. Knowledgeable presenter 


185 


4.82 


.46 


3. Well-prepared presenter 


186 


4.75 


.46 


4. Engaging activities 


182 


4.49 


.76 


5. Want to learn more 


181 


4.57 


.75 


6. Helped me think about learning from student work 


185 


4.62 


.67 


7. Helped me think about using assessment more effectively 


181 


4.60 


.69 


8. Overall worthwhile session 


186 


4.63 


.65 
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Individual session ratings — ^that is, the combined ratings for both offerings of each session — 
tended to be quite high as well, with the lowest mean being 4.16 (SD .76). In five of the individual 
sessions, the highest mean rating concerned the knowledge of the presenter. In the sixth session, the 
presenter’s preparedness received the highest mean rating. The lowest mean ratings for each session 
regarded engaging activities for three sessions, relevance of information for one, desire to learn more 
for one, and worthwhileness for another. But because all the mean scores were above 4 on the 5- 
point scale, it can be concluded that participants nonetheless tended to assess the individual sessions 
favorably (see Appendix C). 

And, in fact, responses to the two open-ended questions on session feedback forms appear 
to confirm that attendees found sessions valuable. In reply to the query concerning what of value 
had been learned, 27 of 33 respondents who had participated in the senior project sessions wrote 
comments such as, “practical — ^how to implement senior projects,” “nuts and bolts of senior project,” 
and “a better imderstanding of what senior projects are and how the long term development has 
improved student learning.” The remaining six respondents did not reply to the query. Forty-three 
participants in the sessions on rubrics completed the individual session feedback form; 21 made 
positive comments and the remainder made no comment. Comments concerning what of value had 
been learned included “strategies for working with teachers to develop rubrics across the 
curriculum,” “affirmed use of rubrics,” and “insights in helping teachers prepare a variety of rubrics 
for a variety of purposes.” Eighteen participants in the sessions on Microsociety completed the 
session feedback form, although only eight answered the question concerning what of most value 
they had learned. Among these eight were, “protocol was thought provoking, good discussions, 
student involvement,” and “learning about Microsociety, using protocol model.” Asked what the 
most valuable things they had learned from participating in the sessions on student writing portfolios 
were, 18 of 36 respondents offered replies such as “practical info on implementation,” “[portfolio] 
offers a lot to individual student,” “how to help son, increasing confidence in public school,” and 
“relevance of portfolio on writing assessment.” The remaining 18 gave no response. Twenty of 37 
respondents to the feedback form concerning the sessions on project-based learning offered 
descriptions of the most valuable things they had learned. These included “info useful in my own 
class,” “[presenter] actually told how projects were organized and implemented,” and “this strategy 
gives student[s] responsibility for learning.” The remainder did not respond. Finally, of the 19 
participants who completed the feedback form concerning the sessions on case discussions, 12 wrote 
of their learnings. They indicated, for instance, having learned about “questioning without threat,” 
“the power of reflecting with peers,” and “questioning strategies and processes is valuable.” 

The feedback forms for individual sessions also asked participants how each session might 
be improved. Twenty-one people who completed the form concerning the sessions on senior 
projects indicated that more time would have been helpful. One respondent suggested, “discuss 
data-where is data that shows implementation of senior projects increases student achievement? Is 
there more than action research, self-assessments . . . ?” Ten participants in the sessions on rubrics 
offered suggestions for improvement, most of which were unique. Some of these were “have 
samples [of rubrics] for high school and elementary [levels],” “examples, bibliography of teacher 
resources,” “not all visuals match book,” “[offer] another session, practice in groups on development 
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of rubrics,” and “consider different experience levels and expertise.” Only one suggestion for 
improvement was offered for the sessions on Microsociety, and this concerned the use of the 
California Protocol; ‘Too structured for my taste. [Use] different process — [it’s] great for 
presenter — ^but as someone thinking of this program — I’d want more logistics.” For this participant, 
then, the Tuning Protocol interfered with learning about Microsociety. Thirteen attendees offered 
suggestions vis-a-vis the sessions on student writing portfolios, several making more than one 
suggestion. Eight participants suggested that the session would have been improved had it offered 
more activities in which attendees practiced assessing student portfolios. Six wrote that they would 
have liked more examples of writing portfolios, two adding that examples from different school 
levels would have been useful. One participant asked, “How can this be used in small schools?” 
Nine suggestions were offered for improving the sessions on project-based learning. Three 
participants requested more time. Three attendees wanted, as one put it, “more hands-on activities 
and less lecture; let participants create project.” One respondent wrote that the presenter had been 
“hard to hear,” while another wanted “more copies of materials.” Another suggested that the 
presenter could have distributed some rubrics for assessing projects. Last, four participants made 
suggestions for improving the sessions on case discussions. One attendee thought the session could 
have been longer. Another wrote, “This session could have been a little more directed . . . The 
presenter clearly knew the content but I needed more crisp pacing and direction.” “More hands-on, 
more practice with a variety of cases,” suggested another. Yet another thought that “role modeling” 
of case discussions would have been valuable. 

Data from the final evaluation form indicate that participants thought the symposium had 
been of high quality. Twenty-six participants completed this form, 14 of whom were Quest network 
members. Asked how they would assess the quality of the event, 25 wrote that the symposium had 
been at least good (the 26th made no comment). Of these, most made comments such as “very 
informative,” “very successful,” and “excellent.” One respondent wrote, “Exceptional. AEL seems 
to have a finger on the pulse of what schools, teachers, and kids need.” 

Participants also conveyed their satisfaction with the symposium during unstructured 
interviews. “I like it . . . especially the content. I think this is a good conference,” said one. Another 
said that she liked “the break out sessions,” although she found the session on rubrics “too 
introductory — too much of a lecture.” She additionally noted that the symposium was more like 
traditional professional development than other Quest events had been. A parent also reported 
finding the symposium valuable, although her son added that the sessions he had attended were 
“geared towards teachers and administrators.” 

Asked to report the most useful things they had learned during the symposium, respondents 
named a variety of topics. Some wrote of having gained valuable information about two topics. 
Nine participants cited learning important information about senior projects; four, about writing 
portfolios; three, about project-based learning; three, about rubrics; and three, about the California 
Protocol. The remaining comments were idiosyncratic. One respondent wrote of, for instance, 
having learned “how to network with others. There are excellent programs in place that we can 
implement in our system.” Another reported gaining “confirmation of what I already do [and] 
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specific ways to tighten my assessment of ‘special projects.’” “Techniques for assessment” had been 
conveyed to yet another attendee. 

Participants also were asked what they liked or disliked about being offered multiple sessions 
fi-om which to choose. Two respondents thought that more sessions needed to have been offered, 
and two other participants did not reply. The remaining 22 reported that they had liked having 
options. “[It was] good to be able to pick what you thought would be of interest and value in [your] 
own situation,” one respondent wrote. “It met the needs of diverse individuals,’ wrote another. 
During an interview, one participant reported that the team from her school “saw all the sessions by 
breaking up,” a strategy which exposed the team to “lots of ideas.” 

Too, attendees were requested to offer suggestions for improving the symposium. Thirteen 
did not reply to this request, but the remaining 13 provided a variety of comments, with some 
offering several suggestions. Three thought some sessions could have benefitted fi-om more time, 
while another suggested that “some sessions should have two parts in order to cover all info.” Two 
respondents suggested that sessions more clearly relevant to elementary level educators should have 
been offered. Other suggestions were unique. One participant wanted more session variety, while 
another simply wanted more topics. “Have [a] session on how Quest works, offered one 
respondent. Two felt that two sessions in the afternoon were too taxing, one adding that perhaps 
three sessions could have been offered in the morning. Another participant thought the symposium 
could have been offered earlier in the summer, and yet another suggested that “time for the school 
teams to meet together and brain storm and complete/compile data” would have been useful. One 
attendee wanted the event to be held in a different hotel. 

At the close of the symposium, participants engaged in a process called Interview Design. 
This activity facilitated the gathering of attendees’ perspectives on various aspects of the 
symposium. Participants were asked what they had learned as a result of attending the symposium. 
Most who answered this question described the benefits they believed were associated with the 
various strategies that had been presented. Other responses were more idiosyncratic. For example, 
“I’ve affirmed my belief that students need to take ownership of their learning” and “Reinforced 
[my] belief that assessment should be based upon instraction . . . Challenged by [the] fact [that] 
assessment should come first — ^planning down.” In regard to project-based learning, one participant 
said, “I see research skills, written communication, and oral presentation skills showcased. I realize 
the depth in which students’ research can be applied through a project.” A second query asked 
attendees what they planned to do back in their schools as a result of participating in the symposium. 
Again, most respondents replied that they hoped to implement one or more of the assessment 
strategies they had learned of during the symposium. One participant said that he intended to 
“revisit senior project. I wanted to eliminate it because it was causing more problems However, I 
want to revisit it and revamp it and incorporate the concepts learned as this workshop.” Another 
reported that she hoped to “fully develop the mini senior project I do now into something more 
meaningful.” Interestingly, several simply said that they planned to use writing more in their 
classrooms. 
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Another question on the Interview Design protocol asked what specific recommendations 
participants would make to their schools regarding the stimulation of continuous improvement. 
Quite a few simply said that they intended to recommend that one or more of the strategies presented 
at the symposium be incorporated. Others were more general, noting their intention, for inst^ce, 
to “concentrate on more organization and purpose in writing,” “involve students more, or stay 
open for ideas and strategies that work elsewhere.” 

Participants in Interview Design were also requested to evaluate the symposium as a whole. 
Asked what the strengths of the symposium had been, respondents said they had enjoyed the variety 
of sessions, the knowledgeable presenters, seeing student work, referring to the symposium 
notebook, doing hands-on activities, networking, and receiving much information in a short amount 
of time. One respondent said, “I liked that the presenters didn’t deal with theory but discussed 
situations they had actually dealt with and knew the ramifications [of]. There was sensitivity to what 
teachers and schools and districts need. There was a broad range of relevant issues but always a 
handle to grasp or understand it in specific ways.” When asked what they liked least about the 
symposium, participants noted that there had been too many sessions, too few sessions, not enough 
time, and most sessions had not been aimed at elementary-level work. 
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CONCLUSIONS AND RECOMMENDATIONS 
Conclusions 

Several conclusions may be drawn based upon data generated by participant observation, 
unstructured interviews, and participant feedback forms. 

In general, most participants thought the symposium had been useful and informative. Many 
reported that the sessions had provided practical assessment strategies that some intended to use back 
in their schools and classrooms. 

Attendees tended to rate the knowledge and preparedness of presenters highly. Participants 
rated the degree to which session activities engaged them actively somewhat less highly, although 
even these scores were quite high. 

Relatedly, several participants suggested that some sessions could have been improved had 
the presenters offered more “hands-on” activities. 

Participants generally liked being offered a variety of sessions. However, one attendee would 
have liked time to meet with her school team. Another suggestion was for a session devoted to 
explaining the Quest project itself. 

Several participants suggested that sessions might have offered material more fully inclusive 
of elementary-level educators. 
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Recommendations 



The evaluator makes several recommendations based upon the data gathered. 

First, Quest staff might consider offering sessions at future symposia that more explicitly 
address the concerns and work of elementary level educators. Alternatively, staff could request that 
presenters provide material at their sessions that better represents elementary, middle, and high 
school-level work. 

Second, because the lowest mean rating that sessions received concerned the degree to which 
they actively involved participants, staff may want to collaborate in the design of sessions to ensure 
that activities are sufficiently engaging. And because some participants expressly suggested that 
sessions could have benefitted from including more hands-on activities. Quest staff might want to 
request that sessions offered at project events integrate at least one such activity. 

Third, although only one attendee commented on the lack of evidence presented regarding 
the effectiveness of the various assessment strategies described in sessions, staff might consider 
asking presenters at future symposia to offer some substantiation for their claims. 

Fourth, participants made a variety of suggestions that staff could consider incorporating in 
other symposia they may offer. These include providing time for school teams to discuss the 
sessions, offering a session on Quest, and scheduling both introductory and advanced sessions to 
accommodate participants’ diverse experience and interest levels. 
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APPENDIX A: 

Individual Session Evaluation Form 
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Name of Session: 



Focus on Student Work: Session Evaluations 
August 10-11, 1998 

At the end of this session, please complete the following quesitons. Your responses will remian 
confidential and anonymous, so please answer as candidly as possible. Your input will help us imporve 
our future symposia. Please return your responses to AEL staff at the end of the session. 

Directions: For each statement, circle the number that best represents your level of agreement. 
1= Strongly Disagree (SD), 2 = Disagree (D), 3 = Neutral (N), 4 = Agree (A), 5 = Strongly 
Agree (SA). If you don’t know, or it doesn’t apply, circle DK. 







SD 


D 


N 


A 


SA 


DK 


1. 


The information presented at this session 
was relevant and useful to me. 


1 


2 


3 


4 


5 


DK 


2. 


The presenter(s) were knowledgeable 
regarding the subject presented. 


1 


2 


3 


4 


5 


DK 


3. 


The presenter(s) were well-prepared and 
well-organized. 


1 


2 


3 


4 


5 


DK 


4. 


The presenter(s) used activities that 
actively engaged me in thinking and 
learning about session topics. 


1 


2 


3 


4 


5 


DK 


5. 


This session stimulated me to want to 
learn more about this subject. 


. 1 


2 


3 


4 


5 


DK 


6. 


This session has helped me think about the 
possibilities of learning from student work. 


1 


2 


3 


4 


5 


DK 


7. 


This session has helped me think about 
how to use assessment more effectively. 


1 


2 


3 


4 


5 


DK 


8. 


Overall, this was a worthwhile, important, 
and informative session. 


1 


2 


3 


4 


5 


DK 



9. What were the most valuable things you learned during this session? 



10. In what ways could this session be improved? 



APPENDIX B: 
Final Evaluation Form 
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Inquiry Into Improvement: Focus on Student Work 
August 10 & 11, 1998 * Gatlinburg, TN 

Your evaluation of this symposium is very important to us as we plan future events. Please take some time to answer the 
following questions. Your responses will remain confidential, so please answer as honestly as possible. Thank you! 

1. How did you learn about this symposium? 



2. Please circle which of the sessions you attended. 

* Senior Projects 

* Using Rubrics to Assess Student Work 

* Micro-society and Tuning Protocol Case 

3. Overall, how would you assess the quality of this 



* Writing Portfolios 

* Project-based learning 

* Methodology for Professional Development 

event? 



4. What was the most useful thing you learned at this symposium? 



5. What did you like about having multiple sessions from which to choose? 



6. What did you dislike about having multiple sessions? 



5. In what ways could the symposium be improved? 



6. In what capacity are you attending this symposium? (Please circle.) 



* Principal 

* Assistant Principal 

* Teacher 

* Other School Staff 



* Community Member 

* Central Office Staff 

* Student 

* Parent 



7. Are you a Quest network member? Yes No 
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Table 2 

Senior Projects Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1 . Relevant and useful information 


33 


4.82 


.73 


2. Knowledgeable presenter 


33 


4.88 


.70 


3. Well-prepared presenter 


33 


4.82 


.73 


4. Engaging activities 


31 


4.71 


.59 


5. Want to learn more 


33 


4.85 


.71 


6. Helped me think about learning from student work 


32 


4.78 


.75 


7. Helped me think about using assessment more effectively 


32 


4.78 


.75 


8. Overall worthwhile session 


33 


4.85 


.71 



Table 3 

Rubrics Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1 . Relevant and useful information 


43 


4.63 


.54 


2. Knowledgeable presenter 


42 


4.79 


.42 


3. Well-prepared presenter 


43 


4.77 


.43 


4. Engaging activities 


43 


4.63 


.66 


5. Want to learn more 


43 


4.49 


.77 


6. Helped me think about learning from student work 


43 


4.63 


.62 


7. Helped me think about using assessment more effectively 


43 


4.63 


.62 


8. Overall worthwhile session 


43 


4.65 


.57 
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Table 4 

Microsociety Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1 . Relevant and useful information 


18 


4.61 


.61 


2. Knowledgeable presenter 


18 


5.00 


.00 


3. Well-prepared presenter 


18 


4.89 


.32 


4. Engaging activities 


17 


4.88 


.33 


5. Want to learn more 


17 


4.82 


.39 


6. Helped me think about learning from student work 


18 


4.89 


.32 


7. Helped me think about using assessment more effectively 


17 


4.76 


.75 


8. Overall worthwhile session 


18 


4.89 


.32 



Table 5 

Student Writing Portfolios Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1 . Relevant and useful information 


36 


4.56 


.50 


2. Knowledgeable presenter 


36 


4.75 


.44 


3. Well-prepared presenter 


36 


4.72 


.45 


4. Engaging activities 


35 


4.26 


.85 


5. Want to learn more 


35 


4.37 


.81 


6. Helped me think about learning from student work 


36 


4.53 


.65 


7. Helped me think about using assessment more effectively 


36 


4.58 


.60 


8. Overall worthwhile session 


36 


4.64 


.59 






Table 6 

Project-based Learning Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1. Relevant and useful information 


36 


4.50 


.65 


2. Knowledgeable presenter 


37 


4.89 


.31 


3. Well-prepared presenter 


37 


4.68 


.53 


4. Engaging activities 


37 


4.27 


.90 


5. Want to learn more 


35 


4.57 


.65 


6. Helped me think about learning from student work 


37 


4.62 


.59 


7. Helped me think about using assessment more effectively 


36 


4.47 


.70 


8. Overall worthwhile session 


37 


4.54 


.65 



Table 7 

Case Discussions Sessions 



Condensed Evaluative Statements 


N 


Mean 


SD 


1. Relevant and useful information 


19 


4.42 


.69 


2. Knowledgeable presenter 


19 


4.58 


.51 


3. Well-prepared presenter 


19 


4.68 


.58 


4. Engaging activities 


19 


4.37 


.83 


5. Want to learn more 


18 


4.39 


.92 


6. Helped me think about learning from student work 


19 


4.26 


.87 


7. Helped me think about using assessment more effectively 


17 


4.29 


.77 


8. Overall worthwhile session 


19 


4.16 


.76 




33 



APPENDIX D: 

Completed Evaluation Standards Checklist 
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Citation Form 



I 

The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

■ request for evaluation plan/design/proposal 

* evaluation plan/design/proposal 

evaluation contract 

I X evaluation report 

other: 

To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 

I Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 



Descriptor 

U1 Stakeholder Identification 


The Standard was 
deemed applicable 
and to the extent 
feasible was taken 
into account. 

X 


The standard was 
deemed applicable 
but could not be 
taken into account. 


The Standard was 
not deemed appli- 
cable. 


Exception was taken 
to the Standard. 


U2 


Evaluator Credibility 


X 








U3 


Information Scope and Selection 


X 








U4 


Values Identification 


X 








U5 


Report Clarity 


X 








U6 


Report Timeliness and Dissemination 


X 








U7 


Evaluation Impact 


X 








FI 


Practical Procedures 


X 








F2 


Political Viability 






X 




F3 


Cost Effectiveness 


X 








PI 


Service Orientation 


X 








P2 


Formal Agreements 


X 








P3 


Rights of Human Subjects 


X 








P4 


Human Interactions 


X 








P5 


Complete and Fair Assessment 


X 








P6 


Disclosure of Findings 


X 








P7 


Conflict of Interest 


X 








P8 


Fiscal Responsibility 


X 








A1 


Program Documentation 


X 








A2 


Context Analysis 


X 








A3 


Described Purposes and Procedures 


X 








A4 


Defensible Information Sources 


X 








A5 


Valid Information 


X 








A6 


Reliable Information 


X 








A7 


Systematic Information 


X 








A8 


Analysis of Quantitative Information 


X 








A9 


Analysis of Qualitative Information 


X 








A10 Justified Conclusions 


X 








All 


1 Impartial Reporting 


X 








A12 Metaevaluation 


X 









I 

I 



Name Caitlin Howley-Rowe 

(typed) 

i-l,,. .. .. ' . 

(signature) 

Position or Title: Research Assistant 

Agency: Appalachia Educational Laboratory 



Date: 9/14/98 
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